Interface scanning for disabled users

ABSTRACT

Systems and processes for scanning a user interface are disclosed. One process can include scanning multiple elements within a user interface by highlighting the elements. The process can further include receiving a selection while one of the elements is highlighted and performing an action on the element that was highlighted when the selection was received. The action can include scanning the contents of the selected element or performing an action associated with the selected element. The process can be used to navigate an array of application icons, a menu of options, a standard desktop or laptop operating system interface, or the like. The process can also be used to perform gestures on a touch-sensitive device or mouse and track pad gestures (e.g., flick, tap, or freehand gestures).

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/785,375 filed on Oct. 16, 2017 and published on Feb. 8, 2018 as U.S.Patent Publication No. U.S. 2018-0039395, which is a continuation ofU.S. patent application Ser. No. 13/782,843 filed on Mar. 1, 2013 andissued on Oct. 17, 2017 as U.S. Pat. No. 9,792,013, which claims thebenefit under 35 USC 119(e) of U.S. Provisional Patent Application No.61/757,052, filed Jan. 25, 2013, the disclosures of which are herebyincorporated by reference in their entirety.

FIELD

This relates generally to electronic devices for physically-impairedusers and, more specifically, to electronic devices that allow amotor-impaired user to navigate a computer interface.

BACKGROUND

Various types of input devices, such as keyboards, mice, track pads,touch screens, and the like, have been developed to enable a user tointeract with a computing device. While these traditional input devicesprovide users with a quick and intuitive way to interact with acomputing device, they all require a significant amount of movement bythe user. This presents a problem for physically-impaired users thatlack the fine motor skills required to operate such input devices. Forexample, some physically-impaired users are unable to move a fingeracross the surface of a touch screen device and thus, cannot operate atouch-sensitive device, such as a tablet computer. Similar problems canarise with the use of a keyboard or mouse.

Thus improved input devices and methods are desired forphysically-impaired users.

SUMMARY

This relates to techniques for scanning a user interface. One processcan include scanning multiple elements within a user interface byhighlighting the elements. The process can further include receiving aselection while one of the elements is highlighted and performing anaction on the element that was highlighted when the selection wasreceived. The action can include scanning the contents of the selectedelement or performing an action associated with the selected element.The process can be used to navigate an array of application icons, amenu of options, a standard desktop or laptop operating systeminterface, or the like. The process can also be used to perform gestureson a touch-sensitive device or mouse and track pad gestures (e.g.,flick, tap, or freehand gestures).

Systems and computer-readable storage media for performing the processesdescribed above are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary scanning system according to variousexamples.

FIG. 2 illustrates an exemplary user interface according to variousexamples.

FIG. 3 illustrates an exemplary grouping of elements of a user interfaceaccording to various examples.

FIG. 4 shows an exemplary hierarchy of elements according to variousexamples.

FIG. 5 illustrates an exemplary process for defining and groupingelements of a user interface according to various examples.

FIG. 6 illustrates an exemplary process for scanning elements of a userinterface according to various examples.

FIGS. 7-51 illustrate exemplary user interfaces showing variousfunctions that can be performed using the process of FIG. 6 according tovarious examples.

FIG. 52 illustrates an exemplary computing system for scanning a userinterface according to various examples.

FIGS. 53-55 illustrate exemplary personal devices that can be used toscan a user interface according to various examples.

DETAILED DESCRIPTION

In the following description of the disclosure and examples, referenceis made to the accompanying drawings in which it is shown by way ofillustration specific examples that can be practiced. It is to beunderstood that other examples can be practiced and structural changescan be made without departing from the scope of the disclosure.

This relates to techniques for scanning a user interface. One processcan include scanning multiple elements within a user interface byhighlighting the elements. The process can further include receiving aselection while one of the elements is highlighted and performing anaction on the element that was highlighted when the selection wasreceived. The action can include scanning the contents of the selectedelement or performing an action associated with the selected element.The process can be used to navigate an array of application icons, amenu of options, a standard desktop or laptop operating systeminterface, or the like. The process can also be used to perform gestureson a touch-sensitive device or mouse and track pad gestures (e.g.,flick, tap, or freehand gestures). Systems and computer-readable storagemedia for performing the processes described above are also disclosed.

FIG. 1 illustrates an exemplary scanning system 100 that allows amotor-impaired user to navigate a computer interface of computing device102. System 100 may generally include a computing device 102communicatively coupled to input device 104. Computing device 102 caninclude any type of computing device, such as desktop computer, laptopcomputer, tablet computer, mobile phone, portable media player, or thelike. Computing device 102 can also include a display for displaying auser interface.

Input device 104 can include any type of input device, such as a trackpad, touch sensitive display, mouse, single-button switch, multi-buttonswitch, camera, infrared (IR) proximity sensor, audio sensor, breathsensor, joystick, or the like, capable of detecting an input from a userindicating that the user is making a selection. For example, inputdevice 104 can include one or more push-buttons having a compressedstate and an uncompressed state, a image sensor configured to detectmovement of a user (e.g., blinking or movement of the user's head), anIR proximity sensor configured to detect an object within a thresholddistance from the sensor, a track pad or touch screen configured todetect the presence of an object (e.g., a user's hand placed on or nearthe surface of the device), an audio sensor configured to detect a soundhaving a predetermined characteristic (e.g., volume, pitch, or thelike), a breath sensor to measure air pressure blown into a straw, apressure sensor to detect depression of a button, an accelerometer todetect movement of a joystick, a sensor to detect the amount ofdisplacement of a joystick, or the like. In some examples, input device104 can be configured to output sensor data to computing device 102where it can be processed to determine if a selection has been made bythe user. In other examples, input device 104 can be configured toprocess the sensor data to determine if a selection has been made by theuser. In yet other examples, both input device 104 and computing device102 can be configured to process the sensor data to determine if aselection has been made by the user.

Input device 104 can be coupled using any wired or wireless technologyto transmit an output signal (e.g., sensor data and/or an indication ofwhether or a not a user selection has been received) to computing device102. For example, a wired connection, such as a USB cable, audio cable,or the like, or a wireless connection, such as Bluetooth, WiFi, NFC, IR,or the like, can be used to communicate between input device 104 andcomputing device 102. In other examples, input device 104 can beincluded within computing device 102. For example, computing device 102can be a tablet computer and input device 104 can be a touch sensor of atouch-sensitive display of the tablet computer.

Computing device 102 can display a user interface having one or moreelements that can be selected by a user (e.g., by hovering over theelement and clicking a mouse button or touching a location of theelement on a touch sensitive display). These selectable elements caninclude, for example, icons corresponding to apps, hyperlinks, items ina menu bar, a section of a display, and the like. To illustrate, FIG. 2shows an exemplary interface 200 having selectable elements (e.g.,application icons) 201-223. Each selectable element 201-223 can beselected (e.g., by touching a location of the element on a touchsensitive display) by a user to perform an action associated with theselectable element. While elements 201-223 are visible within interface200, it should be appreciated that interface 200 can further includeelements that are not visible (e.g., elements that may be displayed inresponse to a particular event, such as receiving a user input).

In some examples, the selectable elements of the user interface can begrouped into sets of one or more selectable elements, which in turn canbe a selectable element itself. The sets of selectable elements can begrouped in any desired fashion to include any number of selectableelements. For example, FIG. 3 illustrates exemplary interface 300showing one possible grouping of the selectable elements of interface200. In particular, each row of selectable elements can be grouped intosets of elements 309, 311, 313, 315, 317, and 319. Each set can betreated as a selectable element that contains sub-elements correspondingto elements 201-223. Thus, with this particular grouping, interface 300can include six elements in the form of sets 309, 311, 313, 315, 317,and 319, with each element corresponding to a different row of elements201-223. These elements can further include their own sub-elements, eachsub-element corresponding to an element 201-223. In the illustratedexample, elements 309, 311, 313, 315, and 319 each include foursub-elements (corresponding to individual elements 201-223), whileelement 317 includes three sub-elements (corresponding to individualelements 201-223). Thus, an element can refer to any item that can beselected within a display. In some examples, the element can includemultiple sub-elements, each of which can also be selected by a user.These sub-elements can also include sub-elements of their own, and soon. This hierarchy structure of elements and sub-elements can beexpanded to any desired level and the grouping at each level of thehierarchy can be performed in any desired manner. Moreover, an elementcan be a sub-element of more than one element.

FIG. 4 illustrates an exemplary view of the hierarchy 400 of elements ofinterface 300. At the top or highest level of hierarchy 400 are elements309, 311, 313, 315, 317, and 319. Each of these elements containssub-elements corresponding to elements 201-223 at the bottom or lowestlevel of the hierarchy. Thus, moving up a level in the element hierarchyrefers to accessing or interacting with a hierarchy level closer to thetop of the hierarchy, while moving down a level in the element hierarchyrefers to accessing or interacting with a hierarchy level closer to thebottom of the hierarchy.

FIG. 5 illustrates an exemplary process 500 for grouping elements withina user interface. At block 501, a plurality of elements within a userinterface can be defined. These elements can be defined in any manner,and can include any selectable items within a display, such as iconscorresponding to apps, hyperlinks, items in a menu bar, a section of adisplay, or the like. At block 503, two or more of the elements can begrouped together to form a set of elements. The elements can be groupedin any desired manner and based on any desired characteristic. Forexample, the elements can be grouped based on location within the userinterface or based on some logical relationship between the elements. Insome examples, the grouping criteria used at block 503 can be selectedautomatically by the device. In other examples, a user can define thegrouping criteria or can manually group elements within a display using,for example, a graphical user interface (GUI) application.

In some examples, process 500 can be performed by the user's computingdevice such that the device can dynamically define and group elementswithin any user interface displayed on the device. In other examples,the user device or another computing device can define and group theelements of a user interface and store the element and groupinginformation in a file associated with the user interface such that anydevices subsequently displaying the interface can access the definedelements and groupings. In yet other examples, content authors candefine and group the elements of a user interface either implicitly(e.g., with the <p> tag in HTML) or explicitly (e.g., with an attributetag to define groupings for scanner software).

Interface 200 shows an example of an interface after block 501 ofprocess 500 has been performed. In this example, 23 elements have beendefined, each corresponding to an element 201-223 displayed within theinterface. Interface 300 shows an example of interface 200 after block503 of process 500 has been performed. In this example, the 23 elementsof interface 200 have been grouped based on location within the userinterface (e.g., based on their location within a row of elements),thereby forming six sets of elements. These six sets of elements canalso be treated as elements themselves, as mentioned above.

FIG. 5 can alternatively be performed by defining the elements in thehighest level of the hierarchy at block 501 (e.g., elements 309, 311,313, 315, 317, and 319), and then dividing these elements intosub-elements (e.g., the individual elements 201-223) at block 503 toproduce the same hierarchically structured set of elements.

As will be described in detail below with respect to FIGS. 6-51,computing device 102 can be configured to “scan” (e.g., iterativelyfocus) through elements displayed within the user interface. Computingdevice 102 can be further configured to select any of the elements inresponse to an input received by input device 104 while the particularelement is being scanned, thereby allowing a user to interact with anyof the elements in the user interface using one or more inputs (e.g.,button press, mouse click, tap on a touch sensitive surface, voicecommand, movement of a part of the users body, or the like). This isparticularly useful for disabled users that are unable to operate inputdevices requiring motion and careful coordination, such as a keyboard,mouse, or touch sensitive display.

FIG. 6 illustrates an exemplary process 600 for scanning throughelements of a user interface displayed by a computing device. Process600 can be performed by any computing device, such as computing device102 of system 100, having or being associated with an input device, suchas input device 104 of system 100. Process 600 can be performed on aninterface containing one or more elements that have been defined and/orgrouped using a process similar or identical to process 500, describedabove. As will be described in greater detail below with respect toblock 603, process 600 can be used to automatically scan throughelements (e.g., scanning is done without user input) of a user interfaceor manually scan through elements (e.g., scanning to the next element isdone in response to a user input) of the user interface.

Process 600 can begin at block 601, where the next element of thecurrent set of elements can be highlighted. In some examples, at thestart of process 600, the current set of elements can be selected to bethe group of elements corresponding to the highest level of the elementhierarchy (e.g., the group of elements that includes all sub-elements)within the currently displayed user interface. For instance, usinginterface 300 shown in FIG. 3 as an example, elements 309, 311, 313,315, 317, and 319 can correspond to the highest level of hierarchy 400since they include all sub-elements (individual elements 201-223) of theinterface.

Additionally, during the initiation of process 600, the next element ofthe current set of elements can be chosen to be any of the elementswithin the current set of elements. For example, the next element can bechosen based on a location of the element within the displayed userinterface (e.g., element closest to the top of the interface, closest tothe left of the interface, or the like). Using interface 300 as anexample, during the initiation of process 600, the initial “nextelement” can be chosen to be element 309 since it is closest to the topof the user interface. In other examples, the elements of the currentset of elements can have an order based on any desired characteristic,such as a location within the user interface, logical ordering, or thelike, and the “next element” can be chosen at block 601 based on thisordering. For instance, one example ordering of elements 309, 311, 313,315, 317, and 319 can be 309, 311, 313, 315, 317, and then 319. Thisexample ordering is based on location of the elements within the userinterface, starting from the top of the user interface and ending at thebottom of the user interface. In other examples, the ordering can beuser-defined (e.g., by the user of the device or content author).

The highlighting of the next element performed at block 601 can includeany change in a visual characteristic of the element or associated withthe element, audible notification, vibration, other tactile cues, or anyother technique to emphasize an element to demonstrate focus. Forexample, to highlight an element, a box can be drawn around the element,a color of the element can be changed, a brightness of the element canbe changed, or a size of the element can be changed.

As an example of the operation performed at block 601, FIG. 7illustrates an interface 700 having an exemplary highlighting 701 of anelement. In this example, the next element was chosen to be element 309of interface 300. However, it should be appreciated that any of theother elements of interface 300 could have been chosen as the initial“next element” used at block 601. As shown in FIG. 7, highlighting 701includes a box drawn around the element. Additionally, an opacity of theuser interface bound by the box of highlighting 701 has been changed.However, as mentioned above, other types of highlighting can be used.

Once the next element of the current set of elements is highlighted, theprocess can proceed to block 603. At block 603, it can be determinedwhether or not a selection has been received while the highlightedelement displayed at block 601 is being displayed. The selection can bereceived by an input device, such as input device 104, and the selectioncan be identified using computing device 102 and/or input device 104, asdiscussed above.

In some examples, process 600 can be used in an automatic scanning modein which block 603 can be performed for a threshold length of time afterthe element is highlighted at block 601 such that any input receivedduring the threshold length of time can trigger a positive determinationat block 603. The threshold length of time can be set to any value andcan, in some examples, be set by the user of the computing device. Forexample, the threshold length of time can be 0.1, 0.2, 0.3, 0.4, 0.5, 1,1.5, 2, or more seconds.

In other examples, process 600 can be used in a manual scanning mode inwhich block 603 can be performed until a command is received from a userinstructing the computing device to scan to the next element. Forexample, a first input from a user (e.g., from a first button on inputdevice 104) can be interpreted as a selection, while a second input fromthe user (e.g., from a second button on the input device 104) can beinterpreted as a “scan next” command, causing termination of block 603.

If it is determined at block 603 that no selection has been received,the process can return to block 601, where the next element of thecurrent set of elements can be highlighted. For example, FIG. 8illustrates an interface 800 having an exemplary highlighting 801 of thenext element in the interface. In this example, the next element wasselected to be subset 311 of interface 300 since it is the next elementin the example ordering provided above. However, it should beappreciated that other orderings of elements can be used to sequentiallyhighlight elements in any desired order. In this way, blocks 601 and 603can be repeatedly performed (in the absence of receiving a selection inan automatic scanning mode or in response to commands from the user toscan the next element in a manual scanning mode) to sequentiallyhighlight each element of the user interface. This can also be referredto as “scanning” the user interface or “scanning” the elements withinthe user interface. In some examples, a threshold number of scanningcycles can be used to limit the number of times a group of elements canbe scanned. After reaching the threshold number of scanning cycles,scanning can be ended. Scanning can resume after a threshold length oftime or in response to a user input. In some examples, the scanning canresume at the highest level of the hierarchy or can resume where it leftoff.

Referring back to block 603 of process 600, if it is instead determinedthat a selection has been received, then the process can proceed toblock 605. At block 605, it can be determined if the selected element(e.g., the element that is highlighted when the selection is received)is associated with an escape command. If it is determined at block 605that the selected element is associated with an escape command, then theprocess can proceed to block 613. As will be discussed in greater detailbelow, the escape command can be used to escape the current hierarchylevel of elements and instead scan the next highest level of thehierarchy. To illustrate the operation performed at block 605, referringto the example shown in FIG. 8, if a selection was made whilehighlighting 801 was being displayed, it can be determined whether ornot the selected element (element 311) is associated with an escapecommand. In this example, the determination returns a negative resultsince element 311 is not associated with an escape command and insteadsimply represents a grouping of elements 205-208. As a result, process600 can instead proceed to block 607.

At block 607, it can be determined whether the selected element (e.g.,the element that is highlighted when the selection is received) includessub-elements. This determination can be accomplished in various ways.For example, the computing device can evaluate the hierarchy structureof the elements (e.g., similar to that shown in FIG. 4) to determine ifthe selected element contains sub-elements. If it is determined at block607 that the selected element contains sub-elements, the process canproceed to block 609. For example, referring to the example shown inFIG. 8, it can be determined whether or not the selected element(element 311) highlighted with highlighting 801 contains sub-elements.In this example, the determination returns a positive result sinceelement 311 includes four sub-elements (e.g., 205-208). Thus, process600 can proceed to block 609.

At block 609, the selected element can be assigned as the current set ofelements (since it contains a set of sub-elements) and the process canreturn to block 601 where an element of the new current set of elementscan be highlighted. Performing block 609 has the effect of moving down alevel in the element hierarchy. Thus, during the next iteration ofprocess 600, the sub-elements of the element previously selected atblock 603 can be sequentially highlighted or scanned. For example, FIG.9 shows highlighting 903 of element 206 of the previously selectedelement 311. In this example, each of the elements (e.g., 205-208) ofelement 311 can sequentially be highlighted by highlighting 903. In someexamples, as shown in FIG. 9, the current set of elements can also behighlighted with highlighting 901. In this example, highlighting 901 issimilar to the previous highlighting 801, except that the box outliningthe element is dashed, rather than solid, and the opacity of the userinterface bound by the box of highlighting 901 remains unchanged.

Referring back to block 607, if it was instead determined that theselected element does not include sub-elements, the process can proceedto block 611. For example, using FIG. 9 as an example, if a selectionwas made while highlighting 903 was displayed around element 206, it canbe determined whether or not element 206 includes subsets of its own. Insome examples, the determination can return a negative result since theelement does not include additional sub-elements. Thus, the process canproceed to block 611. As mentioned above, the elements of an interfacecan include sub-elements that are not displayed. However, for simplicityof discussion, it is assumed that element 206 does not includesub-elements in this example.

At block 611, an action associated with the selected element can beperformed. The action can be any action that was assigned to theselected element. For example, using FIG. 9 again as an example, anaction associated with element 206 can be to open the applicationassociated with element 206.

Thus, blocks 601, 603, 605, 607, 609, and 611 can be viewed asoperations that allow a user to select any selectable element within auser interface by activating one or more inputs (e.g., switch, button,etc.) by causing the computing device to sequentially highlight or scanthe various elements within the display. In response to an activation ofthe single input while an element is highlighted, the computing devicecan interpret the input as a selection of that highlighted element. Inthe case where the highlighted element contains multiple sub-elements,activation of the input can cause the computing device to step down onelevel in the element hierarchy and sequentially highlight or scan thesub-elements of the selected element. This process can be repeated untilthe computing device highlights or scans to the desired element withinthe display. As should be apparent to one of ordinary skill in the art,the hierarchy or grouping of the elements within a display can be doneto quickly navigate to a particular element without having to scanthrough every element within the display. Specifically, by providingvarying levels of scanning granularity, users can quickly navigate downthe element hierarchy to target the desired element. However, it canalso be desirable to scan through each element within a display, inwhich case, no grouping of elements may be used.

While blocks 601, 603, 605, 607, 609, and 611 allow a user to navigatedown the element hierarchy, block 613 can be used to allow a user tonavigate up the element hierarchy. For example, referring back to block605 of process 600, if it was instead determined that the selectedelement is associated with an escape command, the process can proceed toblock 613. For instance, in some examples, an element can include asub-element associated with an escape command that can be interpreted bythe computing device as an instruction to move up a level in the elementhierarchy. To illustrate, FIG. 10 shows highlighting 1001 around theentirety of element 311. This highlighting can occur after each of thesub-elements (e.g., elements 205-208) within element 311 is individuallyhighlighted. In other words, the entirety of element 311 can be thoughtof as a fifth sub-element (that was previously not displayed) of element311. Thus, each of the sub-elements (e.g., elements 205-208) and theentire element 311 can be sequentially highlighted or scanned in acircular fashion. In some examples, this fifth sub-element can beassociated with an escape command to allow the user to navigate up alevel in the element hierarchy to scan through elements 309, 311, 313,315, 317, and 319 again. In this example, if a selection was received atblock 603 while highlighting 1001 was being displayed around element311, the fifth sub-element associated with the escape command becomesthe selected element. Thus, a positive determination can be made atblock 605 and the process can proceed to block 613. Displaying theescape command as a selection of the entire element 311, as shown inFIG. 10, is a space-efficient way of displaying such a command as itdoes not require the display of an additional element that would clutterthe user interface. However, it should be appreciated that using theentire current set of elements as the sub-element associated with theescape command is just one way of displaying the escape element. Inother examples, a separate sub-element or icon (e.g., an icon with an“X,” an icon with the word “Escape,” etc.) associated with the escapecommand can similarly be included within each element to allow the userto navigate up one level in the element hierarchy.

At block 613, the previously used current set of elements can be used asthe new current set of elements. The process can then return to block601 where elements of the newly assigned current set of elements can behighlighted or scanned using blocks 601 and 603. For instance, using theexamples provided above, the current set of elements can includeelements 309, 311, 313, 315, 317, and 319 during the first execution ofprocess 600. Blocks 601 and 603 can cause the elements 309, 311, 313,315, 317, and 319 to be sequentially highlighted. In response to aselection (e.g., in response to a button press or the like) whileelement 311 is being highlighted, element 311 can be assigned as the newcurrent set of elements at block 609. As a result, process 600 can causethe sub-elements of element 311 to be sequentially highlighted at blocks601 and 603 as shown in FIG. 9. In some examples, the sequentialhighlighting of sub-elements can include a highlighting of thesub-element associated with an escape command, as shown in FIG. 10. Inthis example, if a selection is received at block 603 while highlighting1001 was being displayed around element 311, the process can proceed toblock 613, where the previously used “current set of elements” can beused as the new “current set of elements.” In this example, thepreviously used “current set of elements” include elements 309, 311,313, 315, 317, and 319. This has the effect of causing process 600 toagain sequentially scan the elements 309, 311, 313, 315, 317, and 319,as shown in FIGS. 7 and 8.

While specific examples of scanning through sets of selectable elementsof an interface are provided above, it should be appreciated thatnumerous variations can be made. For example, as shown in interface 1100of FIG. 11, one or more of the elements 201-223 can include sub-elementsin the form of a menu of options that is only displayed in response to aselection made while scanning the particular element. In this example, aselection was received while highlighting the element 216, causingprocess 600 to begin scanning the elements of menu 1103 of element 216.Menu 1103 can include various options, such as “Activate,” “Scroll,”“Escape,” press the home button, perform gestures, perform deviceactions, change settings, and the like. It should be appreciated thatany other selectable elements can be included within menu 1103.Additionally, menu 1103 can be further divided into sub-elements (e.g.,two sub-elements). The first sub-element can include the three elementsin the top row, while the second sub-element can include the fiveelements in the bottom row of menu 1103. These elements can be scannedin a manner similar to elements 201-223 described above. For example,interface 1200 of FIG. 12 shows the scanning of the second sub-element1203.

In some examples, the elements of menu 1103 can further includesub-elements of their own. For example, FIG. 13 illustrates some examplesub-element menu 1303 that can be displayed and scanned in response to aselection of a selectable element of menu 1103. In one example, menu1303 can be a “perform device actions” menu and can include variousdevice actions, such as decreasing the volume, increasing the volume,shaking, taking a screenshot, or simulate the physical activation of anyother button or toggle on the device. These elements can be scanned in amanner similar to elements 201-223 and elements in menu 1103, asdescribed above.

FIG. 14 illustrates another exemplary interface 1400 that can be scannedusing process 600, described above. In this example, an element 1401 caninclude sub-elements grouped in any desired manner. For example, eachphysically grouped set of elements (e.g., “About,” “Software Update,”and “Usage”) can be grouped together as a sub-element of element 1401.Each of these sub-elements can be scanned using process 600 as shown ininterface 1500 of FIG. 15. In this illustrated example, sub-element 1501is currently being highlighted. In response to a selection whilesub-element 1501 is highlighted, the sub-elements within 1501 (e.g.,“Auto-Lock” and “Passcode Lock”) can be scanned. As shown in interface1600 of FIG. 16, in response to a selection while the “Passcode Lock”element is highlighted, a menu 1601 similar to that shown in FIGS. 11and 12 can be displayed and scanned.

In some examples, menu 1601 can include a “Scroll” selectable element.In response to a selection received while the “Scroll” selectableelement is highlighted, a scroll menu 1701, shown in interface 1700 ofFIG. 17, can be displayed having various selectable elements. Theseelements can be scanned using process 600, allowing a user to scrolldown, scroll up, scroll right, scroll left, auto scroll, scroll to thetop edge, scroll to the bottom edge, scroll to the left edge, or scrollto the right edge.

For example, the “auto scroll” function associated with the “autoscroll” element can be provided to help the user easily read contentthat is in a scroll area (e.g., vertical or any other direction), acommon occurrence in documents, webpages, and other textual content.This function can cause the content displayed within the user interfaceto scroll at a smooth, continuous pace to allow the user to readlong-form documents without having to manually scroll the content. Thus,in response to a selection received while highlighting the “auto scroll”element, a second scroll menu 1801, shown in interface 1800 of FIG. 18,can be displayed at the bottom of the user interface away from thecontent the user is attempting to read. The elements of scroll menu 1801can be scanned using process 600. In one example, scroll menu 1801 canbe scanned a threshold number of times (e.g., 1, 2, 3, or more) beforebeing removed from the interface. In response to a user input (e.g.,received by input device 104), scroll menu 1801 can be displayed andscanned again. In some examples, the elements in scroll menu 1801 canallow the user to increase/decrease the auto scrolling speed, pause it,scroll to the top edge, scroll to the bottom edge, scroll to the leftedge, or scroll to the right edge.

As mentioned above, process 600 can be used to perform gestures. Inresponse to a selection of a gesture element (e.g., one of the elementsin menu 1203), gesture menu 1903 shown in interface 1900 of FIG. 19 canbe displayed. Gesture menu 1903 can include various selectable elementscorresponding to different gestures that can be performed. Theseselectable elements can be scanned using process 600, allowing a user toperform gestures, such as tap, flick, pinch, pan, tap and hold, doubletap, drag, freehand gesture, or adjust the number of fingers used toperform the gesture. An indicator 1905 can be displayed showing wherethe gesture can be performed. In this example, the empty circle used asindicator 1905 indicates that no selection is currently being made atthe gesture location (e.g., a virtual finger is not held down on thedisplay or a mouse button is not depressed).

In response to a selection while a “Tap and Hold” selectable element ishighlighted, interface 2000 shown in FIG. 20 can be displayed. In thisinterface, the selectable elements (201-223) can be wiggling, indicatingthat the user can move them using a hold and drag gesture. In thisexample, a “Stop” selectable element 2003 can displayed and scannedusing process 600. The previously displayed indicator 1905 can bereplaced with indicator 2005 signifying that a virtual finger is beingheld down on the display (or a mouse button is being depressed with themouse pointer at the location of indicator 2005).

In response to a selection while the “Stop” selectable element 2003 ishighlighted, interface 2100 shown in FIG. 21 can be displayed. Interface2100 can include gesture menu 1903 and can be similar to interface 1900,except the elements 201-223 in interface 2100 can be wiggling due to thepreviously performed “Tap and Hold” gesture. As shown in FIG. 21,indicator 1905 can again be displayed since the virtual finger has beenremoved from the display (or mouse button is no longer depressed).

In response to a selection of a “Drag” selectable element, interface2200 of FIG. 22 can be displayed to allow a selection of the destinationof the “Drag” operation. In this example, interface 2200 can be beendivided into one or more sections 2201 (e.g., nine sections). Thesesections can be formed using process 500, with the sections 2201 beingthe plurality of elements defined at block 501. These sections 2201 canbe scanned using process 600 in a manner similar to that applied toelements 201-223 of the user interface described above. For example, theelements or sections corresponding to the highest level of the hierarchycan be the three rows of sections 2201. These three rows of sections2201 can be divided into sub-elements corresponding to each section2201. Thus, rows of sections 2201 can be scanned and in response to aselection of a row, sections 2201 within a row can be scanned usingprocess 600, as shown in interfaces 2200 and 2300 of FIGS. 22 and 23,respectively.

In some examples, scanning of interface 2200 can begin by displaying thesections 2201 (e.g., nine sections) overlaid on the user interface.Additionally, a selection indicator (e.g., crosshair) can be displayedwithin the center of the grid formed by sections 2201 (at the center ofthe center section 2201). The crosshair can be highlighted, allowing auser to make a selection of the crosshair. If no selection is made (orthe user inputs a command to scan to the next element), the crosshaircan be removed from the display and the rows of sections 2201 can bescanned (e.g., top row, middle row, then bottom row). If no selection ismade while highlighting a row (or the user inputs a command to scan tothe next element), the selection indicator can again be displayed andhighlighted. This scanning rotation can be performed until a selectionis made.

In some examples, in response to a selection of a row of sections 2201,a selection indicator can be displayed within the center of the row ofsections 2201 (at the center of the center section 2201). The crosshaircan be highlighted, allowing a user to make a selection of thecrosshair. If no selection is made (or the user inputs a command to scanto the next element), the crosshair can be removed from the display andthe sections 2201 within the row can be scanned (e.g., left section2201, middle section 2201, right section 2201, then the entire row toprovide a cancel command). If no selection is made while highlighting asection 2201 (or the user inputs a command to scan to the next element),the selection indicator can again be displayed and highlighted. Thisscanning rotation can be performed until a selection is made. In thisexample, the grid for the nine sections 2201 is still displayed after aselection of a row. In other examples, only the selected row may bedisplayed (e.g., as shown in FIG. 25 where only the row of selectedsubsections 2403 is displayed).

In some examples, in response to a selection of a section 2201, section2201 can be further divided into one or more (e.g., nine) sub-elementsor subsections 2403, as shown in interface 2400 of FIG. 24. Thesesubsections 2403 can be scanned in a manner similar to that describedabove for sections 2201 of interface 2200, as shown in interfaces 2400,2500, and 2600 of FIGS. 24, 25, and 26, respectively. This subdivisionof sections can be performed any desired number of times until a desiredelement can be selected. However, in some examples, a minimum subsectionsize can be imposed. In some examples, when the size of the subsectionsfall below a threshold size, a magnifying loupe can be displayed toallow the user to see the elements being selected.

During the grid scanning described above, if a selection is receivedwhile displaying and highlighting the selection indicator (e.g.,crosshair 2605 in interface 2600), the selection can be interpreted as aselection (e.g., mouse click or finger press) made at the center of theindicator.

In response to a selection made while displaying a selection indicator,menu 2703 of interface 2700 shown in FIG. 27 can be displayed. Menu 2703can include various selectable option elements, such as “Drag” and“Cancel,” allowing the user to drag element 216 to the selected locationor cancel the drag operation.

In response to a selection of a “Drag” selectable element, interface2800 shown in FIG. 28 can be displayed. As shown, element 216 is beingdragged to the destination selected in FIG. 26 and element 215 has beenmoved to the previous location of element 216.

In some examples, a scanning settings menu can be provided to the userto customize the scanning performed using process 600. For example, asettings interface 2900 shown in FIG. 29 can include option 2901 toenable or disable the scanning feature, option 2903 to adjust the numberof switches or input devices (e.g., input device 104) being used, option2905 to enable or disable auto scanning (e.g., switch between manual orautomatic scanning), option 2907 to adjust the scanning speed (e.g.,threshold length of time that block 605 is performed in automaticscanning mode), option 2909 for selecting how many scanning cycles areperformed through the elements of a set of elements, option 2911 forselecting the type of scanning (e.g., element scanning, such as thatshown in FIGS. 7-21 and 27-28, or grid scanning, such as that shown inFIGS. 22-26), and option 2913 to enable or disable grouping of elements(e.g., enabling or disabling block 503 of process 500). In someexamples, the computing device can have a default grouping criteria thatis used when performing block 503 of process 500. For example, thedevice can group elements based on location. In other examples, a usercan program the computing device to use desired grouping criteria whenperforming block 503 of process 500. In yet other examples, the user canmanually group the elements of an interface using, for example, a GUIapplication that allows the used to define which elements are to begrouped together.

FIG. 30 illustrates an exemplary settings interface 3000 for assigningswitches to different actions. For example, selection of any of theoptions 3001 can cause a prompt to the user instructing the user toactivate an input (e.g., press a button or switch on an input device) tobe assigned to the action corresponding to the selection option 3001. Asa result, whenever the particular input is activated, the correspondingaction can be performed. This can be a useful feature when the inputdevice used includes more than one button or switch. Options 3001 caninclude various options, such as activate an item, open a notificationcenter, open scanning settings menu, increase volume, stop scanning,move to the next element, home button, etc.

FIG. 31 illustrates an exemplary interface 3100 that illustrates theoperation of process 600 to perform freehand gestures. Interface 3100includes menu 3101 containing various selectable gesture elements. Inresponse to a selection made while a “Freehand” selectable element ishighlighted, interface 3200 shown in FIG. 32 can be displayed. Interface3200 can include indicator 3201 identifying the starting location of thefreehand gesture and a pointing vector 3203 identifying the directionthat the freehand motion will move. In this example, the hollow circleused for indicator 3201 indicates that the virtual finger or pen is notbeing applied to the surface of the display. Interface 3200 can furtherinclude options 3205 that can be scanned using process 600. Theseoptions can include any type of options, such as draw, rotate, drawcurve, auto press (e.g., press down whenever moving a virtual finger),auto lift (e.g., lift virtual finger when movement is stopped), increasemovement speed, decrease movement speed, cancel, etc.

In response to a selection made when a rotate option is highlighted inoptions 3205, interface 3300 shown in FIG. 33 can be displayed.Interface 3300 can include the previously displayed indicator 3201 andpointing vector 3203 (not labeled) as well as turning arrows 3301 and3303 and 90-degree rotation buttons 3305 and 3307. Arrows 3301 and 3303and 90-degree turning buttons 3305 and 3307 can be scanned using process600 to allow a user to select a direction and amount of rotation of thepointing vector.

For example, in response to a selection made while 90-degree turningbutton 3305 is highlighted, interface 3400 shown in FIG. 34 can bedisplayed. As shown in interface 3400, the indicator 3201 (not labeled)has remained stationary, while the pointing vector 3203 (not labeled)has rotated 90 degrees in a clockwise direction. Had 90-degree rotationbutton 3307 been selected instead of button 3305, pointing vector 3203would have rotated 90 degrees in the counter-clockwise direction.

In response to a selection made while turning arrow 3301 is highlighted,interface 3500 shown in FIG. 35 can be displayed. As shown in interface3500, pointing vector 3203 can begin rotating around indicator 3201 inthe direction corresponding to the selected turning arrow. In thisexample, pointing vector 3203 begins and continually turns in theclockwise direction. Interface 3500 can further include “Stop” button3501 that can be selected to stop rotation of pointing vector 3203 oncethe vector is pointing in the desired direction. Alternatively, pointingvector 3203 can rotate only while a selection is received.

In response to a selection made while “Stop” button 3501 is highlighted,interface 3600 shown in FIG. 36 can be displayed. As shown in interface3600, indicator 3201 can be displayed in its stationary position whilepointing vector can also be displayed in a stationary position pointingin a direction based on when the “Stop” button 3501 was selected ininterface 3500. Additionally, options 3205 can again be displayed andscanned using process 600.

In response to a selection made while a move or draw button ishighlighted in options 3205, interface 3700 shown in FIG. 37 can bedisplayed. As shown in interface 3700, a line 3701 can be drawnoriginating from the previous location of indicator 3201 and moving inthe direction indicated by pointing vector 3203. Additionally, a currentposition indicator 3703 identifying the location of the most recentlydrawn portion of line 3701 line can be displayed. The circle withinindicator 3703 can indicate that the virtual finger or pen is currentlybeing applied to the display. This can be caused by the auto pressoption being selected. Interface 3700 can further include “Stop” button3501 that can be selected to stop movement of indicator 3703, therebystopping the drawing of line 3701.

In response to a selection made while “Stop” button 3501 is highlighted,interface 3800 shown in FIG. 38 can be displayed. As shown in interface3800, indicator 3703 can be displayed in its stationary position withpointing vector 3203. Additionally, options 3205 can again be displayedand scanned using process 600.

In response to a selection made while a draw curve button is selected,interface 3900 shown in FIG. 39 can be displayed. As shown in interface3900, line 3701 can be displayed along with indicator 3703. In addition,curve arrows 3901 and 3903 can also be displayed and scanned usingprocess 600.

In response to a selection made while curve arrow 3903 is highlighted,interface 4000 shown in FIG. 40 can be displayed. As shown in interface4000, line 3701 and indicator 3703 can be displayed. Additionally, acurved pointing vector 4003 can be displayed and line 3701 can becontinually drawn along the curve defined by curved pointing vector4003. Interface 4000 can further include “Stop” button 3501 that can beselected to stop drawing line 3701 along the curve defined by curvedpointing vector 4003.

Using the various navigation options (e.g., drawing options) discussedabove, drawings can be formed as shown in interfaces 4100 and 4200 shownin FIGS. 41 and 42, respectively. As shown in interface 4300 of FIG. 43,the auto press option can be deselected to allow movement of indicator3201 without extending line 3701. This allows movement of indicator 3201to the position shown in interface 4400 of FIG. 44. Options 3205 canagain by selected to draw a line as shown in interface 4500 of FIG. 45.

Interface 4600 shown in FIG. 46 illustrates another exemplary use ofprocess 600. In this example, process 600 can be used to scan the keysof a virtual keyboard. As shown in FIG. 46, interface 4600 includes anelement 4601 including multiple selectable virtual keys. This element4601 of selectable virtual keys can be scanned in a manner similar tothat described above with respect to the application icons of interface300. For example, as shown in interface 4700 of FIG. 47, element 4601can be divided by rows into sub-elements including sub-element 4701(containing a row of selectable virtual keys). As shown in interface4800 of FIG. 48, each sub-element 4701 can be further divided intosub-elements of virtual keys containing either the left half or theright half of virtual keys in that row. In some examples, this groupingcan be used as a default grouping. In other examples, a user can definehow the virtual keys are to be grouped. As shown, subset 4801 includesthe virtual keys on the right half of sub-element 4701. As shown ininterface 4900 in FIG. 49, the virtual keys within subset 4801 can bescanned using process 600.

In some examples, a faster scanning mode can be used to scan a keyboard.In these examples, in response to a selection made while a key (e.g.,the “N” key) is highlighted, the input can be interpreted as a selectionof that letter. In other words, a menu, such as menu 1103, may not bedisplayed requiring the user to “Activate” the element. This results infewer selections being made to type a letter. Thus, in response to aselection made while a key (e.g., the “N” key) is highlighted, interface5000 shown in FIG. 50 can be displayed. As shown, the letter “N” hasbeen displayed at the top of the notepad. Additionally, to allow a userto enter a menu that would normally be displayed in response to aselection of an element, an option button 5001 can also be displayed.This button 5001 can be scanned along with the element 4601 of virtualkeys. In response to a selection made while button 5001 is highlighted,interface 5100 shown in FIG. 51 can be displayed. Interface 5100 caninclude menu 5101 containing an “Alternate Key” selectable element thatcan be used to display alternate keys in place of the virtual keys ofset 4601. Menu 5101 can further include options similar or identical tooptions 1203 described above.

As can be seen by the examples provided above, process 600 can be usedto navigate a touch sensitive mobile device, such as a mobile phone,tablet computer, or the like, as well as non-touch sensitive devices,such as laptop computers, desktop computers, and the like. For example,the grid and keyboard scanning options can be used when using a laptopor desktop computer to emulate the use of a standard mouse and keyboard.Similarly, the element or grid scanning options can be used to navigatea touch-sensitive device, as illustrated by the examples provided above.

As should be appreciated by one of ordinary skill in the art, anygrouping of selectable elements, menus, and the like, can be organizedin any desired manner using process 500 to be navigated using process600.

Additionally, in some examples, a user can generate their own panel(e.g., a virtual keyboard) using a GUI application having drag and dropfunctionality. In this way, the user can customize a panel of virtualbuttons capable of being scanned using process 600. Each virtual buttoncan be customized to perform any desired function, such as type aparticular letter, open an application, etc. The size, position, andgrouping of the virtual buttons of the panel can also be adjusted usingthe GUI application.

One or more of the functions relating to scanning a user interface canbe performed by a system similar or identical to system 5200 shown inFIG. 52. System 5200 can include instructions stored in a non-transitorycomputer readable storage medium, such as memory 5203 or storage device5201, and executed by processor 5205. The instructions can also bestored and/or transported within any non-transitory computer readablestorage medium for use by or in connection with an instruction executionsystem, apparatus, or device, such as a computer-based system,processor-containing system, or other system that can fetch theinstructions from the instruction execution system, apparatus, or deviceand execute the instructions. In the context of this document, a“non-transitory computer readable storage medium” can be any medium thatcan contain or store the program for use by or in connection with theinstruction execution system, apparatus, or device. The non-transitorycomputer readable storage medium can include, but is not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus or device, a portable computer diskette(magnetic), a random access memory (RAM) (magnetic), a read-only memory(ROM) (magnetic), an erasable programmable read-only memory (EPROM)(magnetic), a portable optical disc such a CD, CD-R, CD-RW, DVD, DVD-R,or DVD-RW, or flash memory such as compact flash cards, secured digitalcards, USB memory devices, memory sticks, and the like.

The instructions can also be propagated within any transport medium foruse by or in connection with an instruction execution system, apparatus,or device, such as a computer-based system, processor-containing system,or other system that can fetch the instructions from the instructionexecution system, apparatus, or device and execute the instructions. Inthe context of this document, a “transport medium” can be any mediumthat can communicate, propagate or transport the program for use by orin connection with the instruction execution system, apparatus, ordevice. The transport medium can include, but is not limited to, anelectronic, magnetic, optical, electromagnetic or infrared wired orwireless propagation medium.

In some examples, system 5200 can be included within computing device102 and/or input device 104. In these examples, processor 5205 can becoupled to receive an output signal from a sensor of input device 104.As mentioned above, input device 104 can communicate its output usingany wired or wireless communication technique. Processor 5205 can beconfigured to receive the output from a sensor of input device 104 andprocess the output as described above with respect to FIG. 6 and process600. It is to be understood that the system is not limited to thecomponents and configuration of FIG. 52, but can include other oradditional components in multiple configurations according to variousexamples.

FIG. 53 illustrates an exemplary personal device 5300, such as a tablet,that can perform scanning of a user interface according to variousexamples.

FIG. 54 illustrates another exemplary personal device 5400, such as amobile phone, that can perform scanning of a user interface according tovarious examples.

FIG. 55 illustrates an exemplary personal device 5500, such as a laptopcomputer, that can perform scanning of a user interface according tovarious examples.

Therefore, according to the above, some examples of the disclosure aredirected to a method comprising: highlighting a plurality of elementswithin a user interface, wherein at least one of the plurality ofelements comprises a plurality of sub-elements; receiving a selectionwhile the at least one of the plurality of elements is highlighted; andperforming an action on the at least one of the plurality of elements.Additionally or alternatively to one or more of the examples disclosedabove, highlighting the plurality of elements can include changing avisual characteristic of the plurality of elements, generating anaudible notification, or generating a vibration. Additionally oralternatively to one or more of the examples disclosed above,highlighting the plurality of elements within the user interface caninclude sequentially highlighting the plurality of elements within theuser interface. Additionally or alternatively to one or more of theexamples disclosed above, the action performed on the at least one ofthe plurality of elements can include: highlighting a plurality ofsub-elements of the at least one of the plurality of elements.Additionally or alternatively to one or more of the examples disclosedabove, the method can further include: receiving a selection while asub-element of the plurality of sub-elements is highlighted; andperforming an action associated with the sub-element. Additionally oralternatively to one or more of the examples disclosed above,highlighting the plurality of elements within the user interface caninclude: highlighting a first element of the plurality of elements; andhighlighting a second element of the plurality of elements if theselection is not received within a threshold length of time or inresponse to an input from a user. Additionally or alternatively to oneor more of the examples disclosed above, highlighting the plurality ofelements within the user interface can include: ceasing highlighting ofthe plurality of elements in response to highlighting the plurality ofelements a threshold number of times; and resuming the highlighting ofthe plurality of elements after a threshold length of time or inresponse to a user input.

According to the above, other examples of the disclosure are directed toa method comprising: defining a plurality of elements within a userinterface; grouping the plurality of elements into one or more sets ofelements based on at least one characteristic associated with theplurality of elements within the user interface; and scanning theplurality of elements in accordance with the grouping of the pluralityof elements. Additionally or alternatively to one or more of theexamples disclosed above, defining the plurality of elements within theuser interface can include: identifying a plurality of selectableelements within the user interface; and defining the plurality ofselectable elements as an element of the plurality of elements.Additionally or alternatively to one or more of the examples disclosedabove, defining the plurality of elements within the user interface caninclude: dividing the user interface into a plurality of sections; anddefining the plurality of sections as an element of the plurality ofelements. Additionally or alternatively to one or more of the examplesdisclosed above, the at least one characteristic associated with theplurality of elements can include a location of the plurality ofelements within the user interface.

According to the above, other examples of the disclosure are directed toa method comprising: highlighting a first element within a userinterface; determining whether a selection has been received whilehighlighting the first element; if the selection has been received whilehighlighting the first element, performing an action on the firstelement; and if the selection has not been received while highlightingthe first element, highlighting a second element within the userinterface. Additionally or alternatively to one or more of the examplesdisclosed above, the method further includes: determining whether theselection has been received while highlighting the second element; ifthe selection has been received while highlighting the second element,performing an action on the second element; and if the selection has notbeen received while highlighting the second element, highlighting athird element within the user interface. Additionally or alternativelyto one or more of the examples disclosed above, the user interface canbe a user interface of a touch sensitive device, and wherein the firstelement can include a first application icon and the second elementcomprises a second application icon. Additionally or alternatively toone or more of the examples disclosed above, the user interface caninclude a virtual keyboard, and wherein the first element can include afirst virtual key and the second element can include a second virtualkey. Additionally or alternatively to one or more of the examplesdisclosed above, the user interface can include a menu, and wherein thefirst element can include a first option and the second element caninclude a second option.

According to the above, other examples of the disclosure are directed toa method comprising: causing a display of an orientation indicator;causing a display of a plurality of navigation options; highlighting theplurality of navigation options; and causing a display of a positionalindicator moving in a direction of the orientation indicator in responseto receiving a selection while a move option from the plurality ofnavigation options is highlighted. Additionally or alternatively to oneor more of the examples disclosed above, the plurality of navigationoptions can include a plurality of directional options, and wherein themethod can further include: changing the orientation of the orientationindicator in response to receiving the selection while a directionaloption of the plurality of directional options is highlighted.Additionally or alternatively to one or more of the examples disclosedabove, the plurality of directional options can include a clockwise turnbutton and counter-clockwise turn button, and wherein the orientation ofthe orientation indicator can be rotated in a direction associated withthe clockwise turn button or the counter-clockwise button arrow inresponse to receiving the selection while the clockwise turn button orthe counter-clockwise turn button is highlighted, respectively.Additionally or alternatively to one or more of the examples disclosedabove, the plurality of directional options can include a clockwise turnbutton and counter-clockwise turn button, and wherein the orientation ofthe orientation indicator can rotate by a predetermined amount in adirection associated with the clockwise turn button or thecounter-clockwise turn button in response to receiving the selectionwhile the clockwise turn button or the counter-clockwise turn button ishighlighted, respectively.

According to the above, other examples of the disclosure are directed toa non-transitory computer-readable storage medium comprisinginstructions for: highlighting a plurality of elements within a virtualkeyboard, wherein the plurality of elements comprises a set of virtualkeys of the virtual keyboard; receiving a selection while an element ofthe plurality of elements is highlighted; highlighting the virtual keysof the set of virtual keys of the selected element; receiving aselection while a virtual key is highlighted; and causing a display of acharacter associated with the virtual key. Additionally or alternativelyto one or more of the examples disclosed above, the non-transitorycomputer-readable storage medium can further include instructions for:highlighting, prior to highlighting the virtual keys, a first subset ofvirtual keys of the set of virtual keys of the selected element and asecond subset of virtual keys of the set of virtual keys of the selectedelement; and receiving a selection while either the first subset ofvirtual keys of the set of virtual keys of the selected element or thesecond subset of virtual keys of the set of virtual keys of the selectedelement is highlighted, wherein highlighting the virtual keys of the setof virtual keys of the selected element comprises highlighting only thevirtual keys of the subset that was highlighted when the selection wasreceived. Additionally or alternatively to one or more of the examplesdisclosed above, at least one virtual key of the virtual keyboard caninclude an alternate key option. Additionally or alternatively to one ormore of the examples disclosed above, highlighting the plurality ofelements within the virtual keyboard can include sequentiallyhighlighting each of the plurality of elements within the virtualkeyboard for less than 1 second. Additionally or alternatively to one ormore of the examples disclosed above, highlighting the virtual keys ofthe row of virtual keys of the selected element can include sequentiallyhighlighting each of the virtual keys of the set of virtual keys of theselected element for less than 1 second.

Although the disclosure and examples have been fully described withreference to the accompanying drawings, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosure and examples as defined bythe appended claims.

What is claimed is:
 1. A method comprising: at an electronic device witha display and one or more input devices: scanning a plurality ofelements displayed via the display, wherein scanning the plurality ofelements comprises sequentially highlighting the plurality of elements;receiving, via the one or more input devices, a first selection inputwhile scanning the plural of elements; in response to receiving thefirst selection input: in accordance with a determination that the firstselection input is received when a first element of the plurality ofelements is highlighted: displaying, via the display, one or more firstsub-elements proximate to the first element; and scanning the one ormore first sub-elements of the first element; receiving, via the one ormore input devices, a second selection input while scanning the one ormore first sub-elements; and in response to receiving the secondselection input: in accordance with a determination that the secondselection input is received when a respective sub-element of the one ormore first sub-elements is highlighted, performing an action associatedwith the respective sub-element of the one or more first sub-elements.2. The method of claim 1, wherein the one or more first sub-elements aredisplayed in a menu.
 3. The method of claim 2, wherein the one or morefirst sub-elements displayed in the menu are arranged in two or morerows.
 4. The method of claim 2, wherein the menu overlaps a secondelement of the plurality of elements without overlapping the firstelement of the plurality of elements.
 5. The method of claim 1, whereinthe action associated with the respective sub-element comprises:displaying, via the display, one or more second sub-elements of therespective sub-element, different from the first sub-elements; ceasingdisplay of the one or more first sub-elements; and scanning the one ormore second sub-elements of the respective sub-element.
 6. The method ofclaim 5, wherein the one or more second sub-elements of the respectivesub-element are displayed in a menu along an edge of the display.
 7. Themethod of claim 5, wherein the one or more second sub-elements of therespective sub-elements are displayed in a menu proximate to the firstelement.
 8. The method of claim 5, wherein the one or more secondsub-elements comprise one or more gesture sub-elements selectable toperform one or more gestures.
 9. The method of claim 8, wherein the oneor more gestures performable by selection of the one or more gesturesub-elements include a tap gesture, a flick gesture, a pan gesture, apinch gesture, a drag gesture, a tap and hold gesture, a hold gestureand a drag gesture, a double tap gesture, or a free-hand gesture. 10.The method of claim 8, wherein the one or more second sub-elementsinclude a sub-element selectable to adjust a number of fingers used toperform the one or more gestures.
 11. The method of claim 8, furthercomprising: displaying, via the display, when scanning the one or moregesture sub-elements, an indicator of a location at which to beginperforming the one or more gestures.
 12. The method of claim 5, whereinthe respective sub-element comprises a gesture element, wherein the oneor more second sub-elements comprise one or more gesture sub-elementsselectable to perform one or more gestures, and wherein the actionassociated with respective sub-element further comprises displaying, viathe display, an indicator of a location at which to begin performing theone or more gestures.
 13. The method of claim 1, further comprising: inresponse to receiving the first selection input: in accordance with adetermination that the first selection input is received when a secondelement of the plurality of elements, different from the first elementof the plurality of elements is highlighted: displaying, via thedisplay, one or more second sub-elements proximate to the secondelement; and scanning the one or more second sub-elements of the secondelement; receiving, via the one or more input devices, a third selectioninput while scanning the one or more second sub-elements; and inresponse to receiving the third selection input: in accordance with adetermination that the third selection input is received when arespective sub-element of the one or more second sub-elements ishighlighted, performing an action associated with the respectivesub-element of the one or more second sub-elements.
 14. The method ofclaim 8, further comprising: receiving, via the one or more inputdevices, a third selection input while scanning the one or more secondsub-elements; and in response to receiving the third selection input: inaccordance with a determination that the third selection input isreceived when a respective sub-element of the one or more secondsub-elements corresponding to a gesture sub-element including a dragoperation is highlighted, scanning regions of the display to identify alocation at which to end the drag operation.
 15. The method of claim 14,wherein scanning the regions of the display to identify the location atwhich to end the drag operation comprises: scanning a first plurality ofregions of the display to identify a first coordinate in a firstdirection in response to a fourth selection input; and scanning a secondplurality of regions of the display to identify a second coordinate in asecond direction, different from the first direction, in response to afifth selection input.
 16. The method of claim 15, further comprising:after identifying the first coordinate and the section coordinate,displaying, via the display, a visual indicator of the location at whichto end the drag operation corresponding to the first coordinate and thesecond coordinate; and displaying, via the display, one or more thirdsub-elements.
 17. The method of claim 16, further comprising: scanningthe one or more third sub-elements; receiving, via the one or more inputdevices, a sixth selection input while scanning the one or more thirdsub-elements; and in response to receiving a sixth selection input: inaccordance with a determination that the sixth selection input isreceived when a respective sub-element of the one or more thirdsub-elements is highlighted, performing an action associated with therespective sub-element of the one or more third sub-elements, whereinthe action associated with the respective sub-element of the one or moresecond sub-elements.
 18. The method of claim 5, wherein the actionassociated with the respective sub-element further comprises: scrollingone or more of the plurality of elements displayed via the display;wherein the one or more second sub-elements of the respectivesub-element include one or more sub-elements to control operation of thescrolling.
 19. The method of claim 18, further comprising: receiving,via the one or more input devices, a third selection input whilescanning the one or more second sub-elements and while scrolling the oneor more of the plurality of elements; and in response to receiving thethird selection input changing a speed of the scrolling of the one ormore of the plurality of elements.
 20. An electronic device, comprising:one or more processors; a display; one or more input devices; memory;and one or more programs stored in the memory and configured to beexecuted by the one or more processors, the one or more programsincluding instructions for: scanning a plurality of elements displayedvia the display, wherein scanning the plurality of elements comprisessequentially highlighting the plurality of elements; receiving, via theone or more input devices, a first selection input while scanning theplural of elements; in response to receiving the first selection input:in accordance with a determination that the first selection input isreceived when a first element of the plurality of elements ishighlighted: displaying, via the display, one or more first sub-elementsproximate to the first element; and scanning the one or more firstsub-elements of the first element; receiving, via the one or more inputdevices, a second selection input while scanning the one or more firstsub-elements; and in response to receiving the second selection input:in accordance with a determination that the second selection input isreceived when a respective sub-element of the one or more firstsub-elements is highlighted, performing an action associated with therespective sub-element of the one or more first sub-elements.
 21. Anon-transitory computer readable storage medium storing one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device, cause theelectronic device to performing a method comprising: scanning aplurality of elements displayed via the display, wherein scanning theplurality of elements comprises sequentially highlighting the pluralityof elements; receiving, via the one or more input devices, a firstselection input while scanning the plural of elements; in response toreceiving the first selection input: in accordance with a determinationthat the first selection input is received when a first element of theplurality of elements is highlighted: displaying, via the display, oneor more first sub-elements proximate to the first element; and scanningthe one or more first sub-elements of the first element; receiving, viathe one or more input devices, a second selection input while scanningthe one or more first sub-elements; and in response to receiving thesecond selection input: in accordance with a determination that thesecond selection input is received when a respective sub-element of theone or more first sub-elements is highlighted, performing an actionassociated with the respective sub-element of the one or more firstsub-elements.